MEMORANDUM March 22, 2019 


TO: Annie Wolfe 
Officer, Secondary Curriculum and Development 


FROM: Carla Stevens 
Assistant Superintendent, Research and Accountability 


SUBJECT: VOCABULARY.COM: USAGE AND IMPACT ON READING AND ENGLISH 
PERFORMANCE IN HISD, 2017-2018 


Vocabulary.com is an online adaptive game that teaches students words by systematic 
exposure to a variety of question types and activities and by understanding all the word 
meanings and nuances as they are learned. Vocabulary.com teaches 14,000 words using over 
208,000 questions. It has been implemented with the Houston Independent School District’s 
(HISD) fifth through twelfth grade students since the 2014—2015 school year. 


This evaluation used a quasi-experimental design to determine how Vocabulary.com impacted 
students’ performance on the 2018 State of Texas Assessments of Academic Readiness 
(STAAR) Grades 3-8 reading and English 1 End-of Course (EOC) exams. 


Key findings include: 
e The correlation between the number of words students mastered and questions answered 


correctly on Vocabulary.com ranged from 0.78 to 0.96 for the study sample. On average, 
the ratio of questions answered correctly to words mastered was 18:1. 


e The highest proportion of eighth- (97.6%) and ninth-grade (95.7%) students who met the 
Approaches Grade Level standard on the 2018 STAAR 3-8 reading and English | EOC 
exams, respectively, mastered at least 100 words on Vocabulary.com. 


e Most sixth- to eighth-grade students who showed gains (78.0—86.3%) on the 2018 STAAR 
3-8 reading scale scores did not master any words on Vocabulary.com. 


e Treatment effects showed that, on average, sixth- to ninth-grade students in the sample 
who used Vocabulary.com met the Approaches Grade Level standard on the 2018 STAAR 
3-8 reading and English | EOC tests without mastering any Vocabulary.com words. 


e The strongest predictor of students’ performance on Vocabulary.com on the sixth- to eighth- 
grade 2018 STAAR reading was their prior (2017 STAAR 3-8) reading scale score. They 
predicted between 50 to 62 percent of the variance in students’ scale scores. 


Further distribution of this report is at your discretion. Should you have any questions, please 
contact me at 713-556-6700. 


Attachment 


cc: Noelia Longoria 
Jessica Chevalier 


HISD Research and Accountability 2 


iq 


———__ is 
HOUSTON INDEPENDENT SCHOOL DISTRICT 


aE AND IMI j 


ABULARY. Co : USAG 
ON READING AND ENGLISH 


RMA NC =e IN HISD, 2017-201 


V /OC Cc A 


= aaa 


PERF 


Research and Accountability 


ANALY ZING DATA, MEASURING PERFORMANCE 


HISD 


Houston Independent School District 
Hattie Mae White Educational Support Center 
4400 West 18th StreetHouston, Texas 77092-8501 


www.HoustonliSD.org 


It is the policy of the Houston Independent School 
District not to discriminate on the basis of age, color, 
handicap or disability, ancestry, national origin, 
marital status, race, religion, sex, veteran status, 
political affiliation, sexual orientation, gender identity 
and/or gender expression in its educational or 
employment programs and activities. 


2019 BOARD OF EDUCATION 


Diana Davila 
President 


Holly Maria Flynn Vilaseca 
First Vice President 


Elizabeth Santos 
Second Vice President 


Sergio Lira 
Secretary 


Susan Deigaard 
Assistant Secretary 


Wanda Adams 
Jolanda Jones 
Rhonda Skillern-Jones 
Anne Sung 


Grenita Lathan, Ph.D. 
Interim Superintendent of Schools 


Carla Stevens 
Assistant Superintendent 
Department of Research and Accountability 


Ted D. Serrant, Ph.D. 
Sr. Research Specialist 


Venita R. Holmes, Dr.PH. 
Research Manager 


VOCABULARY.COM: USAGE AND IMPACT ON READING 
AND ENGLISH PERFORMANCE IN HISD, 2017-2018 


Executive Summary 


According to its website, Vocabulary.com is an online-based adaptive game that teaches students words 
by systematic exposure to a variety of question types and activities and understanding all the word 
meanings and nuances as they are learned (Vocabulary.com, n.d). Vocabulary.com teaches 14,000 words 
using over 208,000 questions. Fifth- to twelfth-grade students in the Houston Independent School District 
(HISD) have been using Vocabulary.com since the 2014—2015 school year. This report is an evaluation of 
the effect of Vocabulary.com on the reading and English performance of sixth- to twelfth-grade student 
participants. The report, however, focused on the sixth to ninth grades and particularly, the eighth and ninth 
grade, which had the largest number of students in the sample of 3,869. Students who answered at least 
one question on Vocabulary.com, were first-time testers, and had a score on the 2018 State of Texas 
Assessments of Academic Readiness (STAAR) 3-8 or the English | End-of-Course (EOC) regular tests 
were included in the sample. Students’ participation was incentivized with prizes and there were shout-outs 
for high performing students. In addition, students participated in a national Vocabulary Bowl. 


The evaluation used a quasi-experimental design, multivalued treatment effects, descriptive and regression 
analyses, and KonFound-lt© (a software for validating inference) to determine how Vocabulary.com 
impacted students’ performance on the 2018 STAAR 3-8 reading and EOC English | exams. 


Key findings 


e The correlation between the number of words students mastered and questions answered correctly on 
Vocabulary.com ranged from 0.77 to 0.96 for the study sample. On average, the ratio of questions 
answered correctly to words mastered was 18:1. 


e The highest proportion of eighth- (97.6%) and ninth-grade (95.7%) students who met the Approaches 
Grade Level standard on the 2018 STAAR 3-8 reading and English | EOC assessments, respectively, 
mastered at least 100 words on Vocabulary.com. 


e Larger percentages of sixth- to eighth-grade students (78.0—86.3%) who showed gains in their 2018 
STAAR 3-8 reading scale scores did not master any words on Vocabulary.com. 


e Treatment effects showed that, on average, sixth- to ninth-grade students in the sample who used 
Vocabulary.com met the Approaches Grade Level standard on the 2018 STAAR 3-8 reading and 
English | EOC tests without mastering any Vocabulary.com words. 


e Eighth-grade students who mastered at least 100 words on Vocabulary.com, would have, on average, 
met the Masters Grade Level standard, and ninth-grade students who mastered at least one word on 
Vocabulary.com would have, on average, met the Meets Grade Level Standard on the 2018 STAAR 
3-8 reading exams. 


e The strongest predictor of students’ performance on the sixth- to eighth-grade 2018 STAAR 3-8 reading 
exam was their prior (2017 STAAR 3-8) reading scale score, accounting for between 50 to 62 percent 
of the variance in students’ scale scores. Vocabulary.com indicators were not statistically significant 
predictors of STAAR performance. 
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Recommendations 


e Greater effort may have to be made to increase the number of students who consistently use 
Vocabulary.com because when disaggregated, fewer students in the sample had mastered more than 
20 words compared to those who mastered no words or less than 20 words. 


e Classroom observations and participant surveys may be required in future evaluations to determine 
how Vocabulary.com is incorporated into classroom instruction and how it is promoted to achieve the 
kind of sustained use that is necessary for word mastery since, on average, students mastered one 
word for every 18 questions answered correctly. 


e Evaluation assessments that are more compatible with the word content in Vocabulary.com may be 
required to determine its true impact on learning, given that the positive relationship between question 
answered and word mastery is already established, but its correlation to STAAR test scores was weak 
and its predictability on STAAR was not statistically significant. 
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Introduction 


Commenting on Scientifically-Based Reading Research (SBRR), Moats (2007) asserts that “as children 
progress in reading, the variance between good and poor readers is increasingly explained by students’ 
knowledge of vocabulary: the more vocabulary one commands, the more fluid and accurate one’s reading 
knowledge” (p. 16). Low-income preschoolers who come to school without a strong vocabulary experience 
have a lasting disadvantage. They only know about half as many words as their middle-class counterparts 
on starting school and acquire additional vocabulary at a slower rate (Moats, 2007). Moats (2007) 
recommended continued instruction in word meaning tied to content learning and teachers’ own verbal 
habits as strategies for improving student vocabulary. 


The U.S. Department of Education, What Works Clearinghouse (WWC), recommended the provision of 
extensive and varied vocabulary instruction to include in-depth teaching of essential content words, and the 
“use of instructional time to address the meaning of common words, phrases, and expressions not yet 
learned” as they provide strong evidence of vocabulary effectiveness (Gersten, Baker, & Shanahan, et al., 
2007, p. 19 ). This report focuses on English learners. Given the importance of vocabulary in overall 
learning, reading comprehension, and fleuncy, the Houston Independent School District (HISD) has 
provided access to Vocabulary.com as part of its digital resources available to all teachers on the Hub (the 
district's learning management system). 


According to its website, Vocabulary.com is an online platform for introducing students to new words and 
building their vocabulary by systematically exposing them to a wide array of question types, real world 
examples, and activities (Vocabulary.com, n.d.). It combines a “smart” dictionary with an adaptive learning 
game for mastering new words. Vocabulary.com is premised on the belief that students only master a word 
after they have demonstrated an understanding of all the word’s meanings and have encountered it in 
multiple contexts (Vocabulary.com, n.d.). Vocabulary.com serves students from fifth grade to adulthood 
and teaches over 14,000 words using over 208,000 questions and the science of learning to model how 
new words are learned. Vocabulary.com personalizes students’ learning experiences by comparing their 
answers to hundreds of millions of other answers given by other Vocabulary.com readers and choosing the 
best just-in-time questions for users (Vocabulary.com, n.d.). 


In using Vocabulary.com, teachers can create classes and assignments, custom quizzes, and monitor 
students’ progress. Vocabulary.com facilitates school and district wide reporting. Vocabulary.com uses 
computer adaptive testing (CAT) technology to scaffold students’ learning based on their abilities, which is 
determined, as they attempt and complete questions posed. These questions become increasingly difficult 
and complex with success. As a computer adaptive test, Vocabulary.com provides a more realistic picture 
of how well learners know the meaning of the words tested and allows for efficient student testing at different 
levels using different number of items (Vocabulary.com, n.d.). HISD students were exposed to 
Vocabulary.com from the 2014—2015 to 2017-2018 school years. 


The purpose of this evaluation is to report on the usage of Vocabulary.com among HISD students and to 
determine the effect of Vocabulary.com word mastery on students’ reading performance and their 
performance on English | for the 2017-2018 school year. The evaluation was designed to answer the 
following questions: 


1. What were the characteristics of students who participated in Vocabulary.com during the 2017- 
2018 school year? 


2. How did Vocabulary.com participants perform on the 2018 STAAR Assessments? 
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3. What were the effects of Vocabulary.com word mastery on students’ 2018 STAAR 3-8 reading and 
English | EOC test performance? 


4. What were the predictors of reading and English | performance of students with word mastery on 
Vocabulary.com during the 2017—2018 school year? 


Literature Review 


The International Literacy Association (2009) advocated for the use of 21% century technologies in literacy 
instruction. The preponderance of internet access in American schools makes the use of web or computer 
based-technologies a distinct possibility (Dalton & Grisham, 2011). The use of web or computer-based 
technology for vocabulary instruction and learning is referred to as eVoc strategies. eVoc strategies are 
electronic or technology-based strategies teachers can use to develop students’ word and vocabulary 
interests and learning (Dalton & Grisham, 2011). 


Dalton and Grisham (2011) identified ten eVoc strategies that can be used to build vocabulary. Among 
them are (1) learning the visual display of word relationships with text, (2) taking advantage of online word 
reference tools that are also teaching tools, (3) using language translators for just-in-time help for English 
learners (EL), (4) increasing reading volume by reading digital texts, and (5) increasing reading volume by 
listening to digital texts with a text-to-speech tool and audio books. The National Reading Panel suggested 
the use of text-related direct vocabulary instruction, the use of a variety of vocabulary instruction methods, 
including computer-based methods and multiple exposures to words in their varying contexts, among 
others, as key to mastering vocabulary, and enhancing comprehension and reading abilities (Butler, Urrutia, 
Buenger, Gonzalez, Hunt, & Eisenhart, 2010). 


An Iranian study involving 52 Persian-speaking, high school English learners (ELS) was designed to 
compare paper-based (PB) and computer-based (CB) contextualization in vocabulary learning of ELs 
(Ahmadian, Amerian & Goodarzi, 2015). The experimental study assigned 26 students to the PB group and 
26 to the CB group. Using repeated measures pretest, posttest, immediate and delayed posttests, the study 
found that CB contextualization had more effects on vocabulary learning of Iranian ELs than PB 
contextualization of words (Ahmadian, Amerian, & Goodarzi, 2015). The study was confined to one high 
school in a single Iranian province, severely restricting its external validity and generalizability to the larger 
Iranian population of high school students. 


A single school study in the U.S. used students from a tenth-grade English language arts (ELA), a twelfth- 
grade regular level English literature course, and an Advanced Placement (AP) English literature course to 
determine the effect of computer-based self-access learning on weekly vocabulary test scores (Dreyer, 
2014). Ninety percent of the 95 study participants were African American. Participants were observed over 
14 weeks. Students were given weekly vocabulary tests and daily online Quizlet! activities. Students were 
encouraged to use Quizlet in their spare time. There was a strong correlation between the online vocabulary 
review program and short-term vocabulary retention, although most students did not utilize Quizlet and 
were treated as an elective control rather than a treatment group (Dreyer, 2014). Students who paced 
themselves and spread out their study sessions did better than those who used the program last minute as 
a ‘cram’ session. The use of Quizlet also improved the scores of most students, and the multiple use rather 
than the single use of Quizlet for review resulted in higher scores. Students who were proficient in the 
notebook system of learning vocabulary appeared not to need the treatment (Dreyer, 2014). Unlike most 
studies of this type, which tended to focus on English learners in college settings, this study focused on 
high school students in mainstream American classrooms. 


1 Quizlet is a site with online-created study guides and reviews that prepare students for tests and exams. 
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Laufer and Goldstein (2004) studied 435 English learners (EL) to determine, among others, the extent to 
which word knowledge is most closely related to second-language (L2) achievement. The study focused 
on the number of words these learners knew, “vocabulary strength, and a combination of four aspects of 
knowledge meaning that are assumed to constitute a hierarchy of word difficulty: passive recognition 
(easiest), active recognition, passive recall, active recall (hardest)” (Laufer & Goldstein, 2014, p. 399). The 
study results showed that the hypothesized hierarchy was present at all word-frequency levels, and passive 
recall was the best predictor of classroom language performance, “and that vocabulary growth varied with 
different word strength modalities” (Laufer & Goldstein, 2014, 399). 


A study on the effectiveness of computer software in learning a common list of unfamiliar words among 
eleventh-grade students was conducted by comparing the program’s instructional approach that 
incorporates definitional and contextual word knowledge with approaches that use definitional and 
contextual information only (Kolich, 1991). The results showed that students who used a modified mixed 
approach that provided sentence context, definitional or synonyms clue, and optional word choices learned 
more words than did students who received definitional information only (Kolich, 1991). 


Two groups of 300 randomly-assigned participants were used to measure vocabulary retention from web- 
based and paper-based learning among Iranian foreign-language learners (Gorjian, 2012). Gorjian (2012) 
administered two-quizzes, two weeks apart, based on twelve expository passages to determine short- and 
long-term retention effect of learning vocabulary. Findings indicated that there was a significant effect of 
the Web-Based Language Learning (WBLL) on short-term vocabulary retention; however, posttest results 
showed that in the long-term, the treatment effect faded (Gorjian, 2012). 


Overall, there appear to be strong and significant links between vocabulary learning and web- or computer- 
based approaches to learning. It appears, that in the long term, the vocabulary retention effects fade. It also 
appears that more frequent use of eVoc web-based or computer-based strategies tend to result in better 
vocabulary outcomes even though studies tended to be foreign-based and focused on English language 
learners. However, few studies appeared to link vocabulary to students’ performance on standardized tests. 
This study will focus on U.S. middle and high-school students in one urban school district using state 
standardized tests. 


Method 


This is a quasi-experimental study designed to determine the effects of Vocabulary.com word mastery on 
students’ reading and English | performance. Student results on the State of Texas Assessments of 
Academic Readiness (STAAR) Grades 3-8 reading and English | End-of-Course (EOC) tests were used 
as the performance measures. Initially, students are graded based on the number of correct items (raw 
score) on each test. Raw scores are standardized and reported as scale scores. Cut scores are determined 
to identify students who met Approaches, Meets, or Masters Grade Level standards on the STAAR tests. 
STAAR neither administers a vocabulary assessment nor disaggregates scores to demonstrate student’s 
vocabulary performance. Notwithstanding, the assumption is that words students encountered in 
Vocabulary.com are possibly similar to those contained in the STAAR tests. However, the correlation 
between students’ mastery of vocabulary and STAAR 3-8 reading/English | EOC results in this study 
sample was weak, although significant. 


Data Collection 


Vocabulary.com recorded the number of questions students answered correctly and the number of words 
they mastered. A list of students who used vocabulary.com was obtained from the program vendors through 
the relevant HISD curriculum manager. The data were transmitted via email using a password protected 
Microsoft Excel spreadsheet. The dataset included the number of questions answered correctly by each 
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student, and the number of words each student mastered. Students were expected to answer about 20 
questions correctly to master one word on Vocabulary.com. 


A total of 29,066 students made up the 2017-2018 dataset from Vocabulary.com. Only students who had 
answered at least one question correctly were included in the analysis. On average, students answered 
932 questions correctly and mastered 52 words. 


Students’ educational and demographic data were obtained from the Public Education Information 
Management System (PEIMS) and merged with the 2017-2018 STAAR 3-8 and EOC performance data 
and Vocabulary.com data unique identifiers. Students in the sample were first-time testers who took the 
STAAR tests. Data for sixth-, seventh-, eighth-, and ninth-grade students were used for the analyses since 
they represented the largest group of students in the sample, which, when disaggregated, would be 
sufficient to produce reportable results (five or more participants). Additionally, most first-time testers who 
took the 2018 STAAR English | EOC exams were enrolled in the ninth grade. The sample comprised of 
3,869 students (2,150 ninth- to twelfth-grade students and 1,719 sixth- to eighth-grade students). Statistical 
power test using an online sample size calculator indicates that a total of 379 students would be adequate 
to make inferential statements of the Vocabulary.com population using a confidence level of 95 percent and 
a confidence interval (Cl) of 5. 


Data Analysis 


The data analyses sought to describe the performance of students with word mastery on Vocabulary.com 
and determine the effects of the number of Vocabulary.com words students mastered on students’ reading 
or EOC English | performance on the STAAR 3-8 and EOC assessments. Preliminary review of the data 
showed strong correlation by grade between the number of questions students answered correctly, and the 
number of words they mastered in Vocabulary.com. The evaluation used a multivalued treatment model to 
determine the effect of Vocabulary.com word mastery on students’ scale scores. Multivalued treatment 
models are useful where there are more than two treatment or intervention levels. The treatment was 
restricted only to students who had mastered at least one word in Vocabulary.com. Those who did not 
master any words were treated as the comparison or untreated group. Because of the wide range in the 
number of words mastered (O—4739), the treatment was divided in seven levels (0 to 6) based on the 
number of words mastered: O = 0; 1 = 1-19; 2 = 20-39; 3 = 40-59; 4 = 60-79; 5 = 80-99; and 6 = 2 100. 
The data were analyzed using multivalued treatment effects without control for other demographic variables 
since convergence was not possible on models when disaggregated by grade indicating insufficient data 
at that level to model the outcome after adjustments. 


The average treatment effect on the treated (ATET) was reported as a measure of how students who 
mastered at least one Vocabulary.com word performed, on average, compared to if they had not mastered 
any words (potential outcome means). The study assumed that word mastery would have a positive effect 
on STAAR 3-8 reading and English | EOC assessments. This is based on the premise that the words 
mastered, and outcome scale scores were adequately related. The evaluation also predicted students’ 
performance on the Vocabulary.com who had test results using multiple linear regression. Students’ 2017 
STAAR 3-8 reading scores, Vocabulary.com variables, and key demographic and educational variables 
were used in the regression models. 


Finally, the study used KonFound-It© (Frank, Maroulis, Duong, Kelcey, 2013), an online software for 
validating confounding inferences. The largest statistically significant predictors in the regression models 
were used to test these inferences. KonFound-It© uses the regression coefficient, the standard error, 
sample size, p-value (0.05), and the number of covariates in a regression model to determine if the 
inference regarding a predictor is valid and what would be required to invalidate that inference. The higher 
the percentage of cases required for invalidation, the more robust the inference and validity. Konfound.It© 
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also indicates what it would take for an omitted variable, if added to the model, to change the validation in 
linear regressions (Frank, 2000). 


The data were tested for and met normality, homoscedasticity, and collinearity conditions using the Shapiro- 
Wilk test, the normal Q-Q plot, and the Detrended normal Q-Q plot on the SPSS statistical analysis software. 
Data were presented using summary tables and charts. 


Limitations 


e Because schools, classrooms, and students self-selected to use Vocabulary.com, finding a comparable 
group of non-users with equal motivation to use the platform was not feasible. As a result, the students 
who did not master any words were used as a comparison or “untreated” group to determine the effect 
of Vocabulary.com on students’ reading and English | performance. 


e STAAR reading and English | assessments do not report vocabulary performance. Their usefulness in 
determining Vocabulary.com effect was limited. PSAT and SAT word-in-context scores were 
considered as alternatives but, similarly, showed weak correlations. 


e The challenge of obtaining convergence on the maximum likelihood indicators between words 
mastered and STAAR outcomes limits the robustness of the analyses, particularly, when the outcome 
is modeled using key demographic and educational variables. The results, therefore, should be 
interpreted with caution. Konfound.It© was used to ensure robustness and validity of inferences. 


Results 


What were the characteristics of students who participated in Vocabulary.com during the 2017- 
2018 school year? 


Table 1 shows the demographic and educational characteristics of the evaluation sample. 


Table 1. Demographic and Educational Characteristics of the Evaluation Sample, 2017-2018 


thibur Sixth Grade Seventh Grade Eighth Grade Ninth Grade 
siete n=77 % n= 139 % n= 1,503 % n=975 % 

Gender Female 38 49.4 70 50.4 825 54.9 518 53.1 

Male 39 50.7 69 49.6 678 45.1 457 46.9 
Econ. No 36 46.8 45 32.4 742 49.4 398 40.8 
Disadv. 

Yes 41 53.3 94 67.7 761 50.6 577 59.2 

No 49 63.6 66 47.5 779 51.8 429 44.0 
At-Risk 

Yes 28 36.4 73 52.5 724 48.2 546 56.0 
Special No 70 90.9 129 92.8 1,413 94.0 921 94.5 
Education 

Yes 7 9.1 10 7.2 90 6.0 54 5.5 

No 63 81.2 118 84.9 1,336 88.9 838 86.0 
LEP 

Yes 14 18.2 21 15.1 167 11.1 137 14.1 

Asian 3 * 5 3.6 131 8.7 71 7.3 

a African 

Ethnicity jrerican 13 16.9 26 18.7 387 25.8 131 13.4 

Hispanic 50 64.9 80 57.6 671 44.6 609 62.5 

White 8 10.4 27 19.4 286 19.0 142 14.7 

No 34 44.2 86 61.9 859 57.2 2 : 
GIT Yes 43 55.8 53 38.1 644 42.9 : : 


Source: Vocabulary.com (data only). 
*Denotes less than 5 students. 
Note: Figures may exceed 100 percent due to rounding. 
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e Most students in the evaluation sample were Hispanic (44.6—64.9%). In addition, most students were 
economically-disadvantaged (50.6—67.7%). 


e A higher proportion of gifted and talented (G/T) students in the sixth grade (55.8%) compared to their 
non-G/T counterparts (44.2%) comprised the sample. 


e There was a higher proportion of non-at-risk students compared to their at-risk peers who comprised 
the sample in sixth and seventh grade (63.6 and 51.8%, respectively). The opposite was true for 
seventh and ninth grades. 


e There were lower percentages of students with limited English proficiency among all grades in the 
sample (11.1—18.2%). Special education students in the sample ranged from 5.5 to 9.1 percent. 


Table 2 shows the percentage distribution of students by grade who mastered Vocabulary.com words 
during the 2017-2018 school year. 


Table 2. Percentage Distribution of Words Mastered on Vocabulary.com by Grade, 2017-2018 


Sixth Seventh Eighth Ninth Tenth Eleventh Twelfth 
Words Grade Grade Grade Grade Grade Grade Grade 
Mastered n % n % n % n % n % n % n % 
0 46 59.7 91 65.5 593 39.5 459 47.1 214 55.4 260 52.7 149 55.2 
1-19 23 29.9 24 17.3 359 23.9 249 25.5 108 28.0 152 30.8 82 30.4 
20-39 2 * 5 3.6 116 7.7 85 8.7 37 9.6 35 7.1 23 8.5 
40-59 2 * 3 . 70 4.7 42 4.3 8 2.1 17 3.5 7 2.6 
60-79 2 * 4 = 46 3.1 28 2.9 4 * 12 2.4 2 * 
80-99 - - 1 i 23 1.5 18 1.9 10 2.6 9 1.8 2 * 
2100 2 * 11 7.9 296 19.7 94 9.6 5 1.3 8 1.6 5 1.9 
Total GE 139 1,503 975 386 493 270 


Source: Vocabulary.com (data only). 
*Denotes less than five students. 
Note: Figures may exceed 100 percent due to rounding. 


e Most Vocabulary.com participants in the sample were enrolled in the eighth (1,503) and ninth-grades 
(975). 


e Most students in the sample by grade (39.5—65.5%) did not master any words on Vocabulary.com, 
followed by students who mastered 1-19 words (17.3-—30.8%). 


e About 20 percent of eighth-grade students mastered at least 100 words, followed by 9.6 percent of 
ninth-grade students. 


e Onaverage, students answered 932 Vocabularly.com questions correctly and mastered 52 words. This 
is equivalent to a ratio of approximately 1:18. 


e The correlation between words mastered and questions answered correctly ranged from 0.82 to 0.95 
for sixth to eighth grade and 0.77 to 0.96 for ninth to twelfth grades. 
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How did Vocabulary.com participants perform on the 2018 STAAR Assessments? 


Figure 1 shows eighth-grade Vocabulary.com students’ performance on the 2018 STAAR 3-8 reading 
exam. Details for sixth and seventh grades are in Table 3, p. 16. 


Figure 1. Percentage of Eighth-Grade Students Who Met Approaches Grade Level Standard on 
STAAR 3-8 Reading by the Number of Vocabulary.com Words Mastered, 2017-2018 
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e The highest percentage of eighth-grade students in the sample (97.6%) who met the Approaches Grade 
Level standard on the 2018 STAAR 3-8 reading was those who mastered at least 100 words on 
Vocabulary.com. 


e Students in the sample who mastered 1-19 words on Vocabulary.com had the lowest proportion of 
students (79.7%) who met the Approaches Grade Level standard on the 2018 STAAR 3-8 reading test. 


e According to Figure 1, 81.2 percent of eighth-grade students who did not master any words on 
Vocabulary.com met the Approaches Grade Level standard on the 2018 STAAR 3-8 reading test. 


Figure 2 shows the performance of ninth-grade Vocabulary.com students on the 2018 STAAR English | 
EOC exam. 


Figure 2. Percentage of Ninth-Grade Students who met the Approaches Grade Level Standard on 
STAAR English | EOC by the Number of Vocabulary.com Words Mastered, 2017-2018 
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e The percentage of ninth-grade Vocabulary.com students in the sample who met Approaches Grade 
Level standard increased progressively from those who did not master any words to those who 
mastered at least 100 words. 


e The highest percentage of ninth-grade students in the sample (95.7%) who met the Approaches Grade 
Level standard on the 2018 STAAR English | EOC was those who mastered at least 100 words on 
Vocabulary.com. 


e The lowest proportion of ninth-students in the sample (62.3%) who met the Approaches Grade Level 
standard on the 2018 STAAR 3-8 reading test did not master any words on Vocabulary.com. 


Table 4 (p. 16) displays the distribution of gains in reading scale scores by the number of words students 
in the sample mastered on Vocabulary.com. 


e The largest percentage of sixth (78.3%) and eighth grade (86.3%) students who made gains on their 
2018 STAAR 3-8 reading scale scores from the previous year did not master any words on 
Vocabulary.com. 


e The largest percentage of seventh-grade students (87.5%) who made scale score gains on the 2018 
STAAR 3-8 reading mastered between 1 and 19 words on Vocabulary.com. 


What were the effects of Vocabulary.com word mastery on students’ 2018 STAAR 3-8 reading and 
English | EOC test performance? 


Multivalued treatment effects with regression adjustment analysis was used to determine the treatment 
effect (ATET) of Vocabulary.com on students’ STAAR reading and English | EOC test performance. As 
mentioned earlier, the intervention variable (words mastered) was divided into six groups of 20 words each. 
Table 5 to Table 8 (pp. 17-18) display the effects of Vocabulary.com words mastered on the 2018 STAAR 
3-8 sixth-, seventh-, and eighth-grade reading and ninth-grade English | EOC tests. 


Sixth Grade 


e Had students not mastered any words (potential outcome means (POM)) on Vocabulary.com, they 
would have, on average, attained a statistically significant mean reading scale score of 1598.1 (p < .05, 
one-tailed) or they would have met, on average, the Approaches Grade Level standard on the 2018 
STAAR 3-8 reading exam (Table 5, p. 17). 


e Onaverage, only students who had mastered 60-79 (4 vs. 0) words on Vocabulary.com demonstrated 
a Statistically significant increase (164.93 scale score points (Ssp)) in their reading scale scores above 
that of students who would not have mastered any words (1598.1 ssp), p < .05 (one-tailed) (Table 5, 
p. 17). These students, on average, would have performed at the Masters Grade Level standard. 


Seventh Grade 


e Had seventh-grade students in the sample not mastered any words (POM) on Vocabulary.com, they 
would have, on average, attained a statistically significant mean reading scale score of 1673.6 (p < 
.001 (one-tailed)). On average, they would have also met the Approaches Grade Level standard on the 
2018 STAAR 3-8 seventh-grade reading test (Table 6, p. 17). 


e Onaverage, only students who had mastered 80—99 words (5 vs. 0) on Vocabulary.com demonstrated 
a statistically significant increase (243.4 scale score points (Ssp)) in their average reading scale score 
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above that of students who would not have mastered any words (1673.6 ssp), p < .05 (one-tailed) 
(Table 6, p. 17). On average, these students would have met the Masters Grade Level standard on the 
2018 STAAR 3-8 seventh-grade reading exam (Table 6, p. 17). 


Eighth Grade 


e Had eighth-grade students in the sample not mastered any words (POM) on Vocabulary.com, they 
would have, on average, attained a statistically significant mean reading scale score of 1688.0 (p < 
.001 (one-tailed)). On average, they would have also met the Approaches Grade Level standard on the 
2018 STAAR 3-8 seventh-grade reading test (Table 7, p. 17). 


e On average, all students except those who had mastered 80-99 words (5 vs.0) on Vocabulary.com, 
demonstrated a statistically-significant increase in their 2018 STAAR 3-8 reading scale scores above 
that of students who would not have mastered any words (1688.0 ssp) (Table 7, p. 17). On average, 
these eight-grade students would have met the Meets Grade Level standard. 


e Eighth-grade students who mastered at least 100 words had the largest statistically significant increase 
(144.5 ssp) (p < .001), which would have placed them, on average, at the Masters Grade Level 
standard, followed by students who mastered 60—79 words (53.8 ssp) (p < .05), and students who 
mastered 20-39 words (43.2 ssp) (p < .05), (Table 7, p. 17) on the 2018 STAAR 3-8 reading test, 
which would have placed them, on average, at the Meets, Grade Level standard. 


Ninth Grade 


e Had ninth-grade students in the sample not mastered any words (POM) on Vocabulary.com, they would 
have, on average, attained a statistically significant mean reading scale score of 3969.8 (p < .001, one- 
tailed). On average, they would have also met the Approaches Grade Level standard on the 2018 
STAAR English | EOC exam at the ninth grade (Table 8, p. 18). 


e On average, all ninth-grade students, who mastered at least one word on Vocabulary.com, 
demonstrated a statistically significant increase in their 2018 STAAR English | EOC scale scores above 
that of students who would not have mastered any words (3969.8 ssp), (Table 8, p. 18). 


e Ninth-grade students who mastered at least 100 words (6 vs.0) had the largest statistically significant 
increase (687.7 ssp) (p < .001), followed by students who mastered 80—99 words (5 vs.0) (526.8 ssp) 
(p < .001), and students who mastered 60-69 words (4 vs. 0) (314.0 ssp) (p < .05), (Table 8, p. 18) on 
the 2018 English | EOC test. 


What were the predictors of reading and English | performance of students with word mastery on 
Vocabulary.com during the 2017-2018 school year? 


Table 9 to Table 12 (pp 19-20) predicted the STAAR performance of sixth, seventh, eighth, and ninth- 
grade Vocabulary.com students in the sample. Eight predictors were used in each model and the Beta was 
reported as a standardized measure for ease of comparison. The eight predictors were Vocabulary.com 
words mastered, and questions answered correctly; economically disadvantaged, at risk, special education, 
limited English proficiency (LEP), G/T, and STAAR prior reading scores where they exist. G/T designation 
was not reported as a ninth-grade predictor. The KonFound-It© software was used to validate the inference 
of the strongest predictor in each model. 
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Sixth Grade 


Overall, the sixth-grade model predicted about 85.3 percent of the variance in the reading scale scores 
of sixth-grade Vocabulary.com students in the sample (Table 9, p. 19). 


None of the two Vocabulary.com predictors were statistically significant predictors of students’ 
performance on the 2018 STAAR 3-8 sixth-grade reading exam (Table 9, p. 19). 


There were two statistically significant predictors. Students’ prior (2017 STAAR 3-8) reading score 
predicted 62 percent of the variance in their performance and being enrolled in special education 
predicted 18 percent of the variance in their performance on the STAAR 3-8 sixth-grade reading test. 
Special education was a negative predictor (Table 9, p. 19). 


As shown, students’ prior reading score predicted 62 percent of the variance in sixth-grade reading 
performance. To make this prediction invalid, 70 percent of this estimated prediction would have to be 
biased and 70 percent (54) of the sixth-grade students in the sample would have to be replaced with 
students for whom the effect of their prior reading score is zero. 


Seventh Grade 


Overall, the seventh-grade model predicted 71.2 percent of the variance in the 2018 STAAR 3-8 
reading scale scores of seventh-grade Vocabulary.com students in the sample (Table 10, p. 19). 


Neither of the two Vocabulary.com predictors were statistically significant predictors of students’ 
performance on the 2018 STAAR 3-8 seventh-grade reading exam (Table 10, p. 19). 


There were three statistically significant predictors. Students’ prior reading score predicted 50% of the 
variance in the 2018 STAAR 3-8 seventh-grade reading performance, G/T predicted 24 percent, and 
LEP predicted 14 percent of the variance. LEP was a negative predictor (Table 10, p. 19). 


As noted, students’ prior reading score predicted 50 percent of the variance in seventh-grade reading 
performance. To make this prediction invalid, 72 percent of this estimated prediction would have to be 
biased and 72 percent (100) of the seventh-grade students in the sample would have to be replaced 
with students for whom the effect of their prior reading score is zero. 


Eighth Grade 


Overall, the eighth-grade model predicted 69.2 percent of the variance in the 2018 STAAR 3-8 reading 
scale scores of eighth-grade Vocabulary.com students in the sample (Table 11, p. 20). 


Neither of the two Vocabulary.com predictors were statistically significant for students’ performance on 
the 2018 STAAR 3-8 eighth-grade reading exam (Table 11, p. 20). 


There were six statistically significant predictors. Students’ prior reading score predicted 56 percent of 
the variance in the 2018 STAAR 3-8 eighth-grade reading performance, at-risk predicted 16 percent, 
special education predicted 11 percent, G/T predicted nine percent, and limited English proficiency and 
economically disadvantaged predicted five percent, each of the variance. G/T and prior reading scores 
were positive predictors (Table 11, p. 20). 
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e As noted, students’ prior reading score predicted 56 percent of the variance in eighth-grade reading 
performance. To make this prediction invalid, 93 percent of this estimated prediction would have to be 
biased and 93 percent (1,392) of the eighth-grade students in the sample would have to be replaced 
with students for whom the effect of their prior reading score is zero. 


Ninth Grade 


e Overall, the ninth-grade model predicted about 60.0 percent of the variance in the 2018 STAAR English 
| EOC scale scores of Vocabulary.com students in the sample (Table 12, p. 20). 


e There were four statistically significant negative predictors. Being at risk predicted 55 percent of the 
variance in the 2018 STAAR ninth-grade English | EOC performance, LEP predicted 22 percent, special 
education predicted 15 percent, and economically disadvantaged status predicted 12 percent of the 
variance (Table 13, p. 20). 


e As noted, being at risk predicted 55 percent of the variance in the ninth-grade reading performance. To 
make this prediction invalid, 92 percent of this estimated prediction would have to be biased and 92 
percent (893) of the ninth-grade students in the sample would have to be replaced with students for 
whom the effect of being at-risk is zero. 


Discussion 


Several things stood out with respect to the effect of Vocabulary.com word mastery on students’ STAAR 
3-8 reading and English | EOC test performance. The data showed that 81.2 percent of eighth-grade and 
62.3 percent of ninth-grade students in the sample who did not master any words on Vocabulary.com met 
the Approaches Grade Level standard on the 2018 STAAR 3-8 reading tests. Students may not have 
required Vocabulary.com word mastery to meet the lowest level of passing on the STAAR exams. However, 
mastering more than 100 words was associated with a passing rate of 97.6 percent and 95.7 percent, 
respectively. 


Further analysis indicated that neither the number of words mastered, nor questions answered correctly, 
were significant predictors of students’ performance. Possible explanations include the weak correlations 
between the number of words mastered or questions answered correctly on Vocabulary.com and STAAR 
test results used in the study. Additionally, STAAR tests are designed to assess grammar and 
comprehension rather than vocabulary. Unless there is compatibility or overlap between the words in the 
predictor variable and the outcome variable, it may not be possible to obtain significant data congruence. 
Another issue may be retention. When students’ short and long-term vocabulary retention were tested, 
results showed that long-term effects faded (Gorjian, 2012). Dreyer (2014) found similar short-term 
retention in his vocabulary intervention study of high school students in the U.S. Further, effective 
vocabulary intervention appeared to be associated with the use of multiple quizzes, student pacing, 
repeated use of the software (Dreyer, 2014), and in this case, word mastery, as postulated earlier. 


Analyses of the effects of Vocabulary.com on students’ STAAR reading and English | EOC performance 
suggested that students exposed to the software, that is, who answered questions correctly, on average, 
would have met the Approaches Grade Level standard. Data on their exposure to or utilization of the 
software were not available nor was information on how the software was incorporated in the classroom 
reading or English | instruction. Kolich’s (1991) study found that students who used a modified mixed 
approach that provided sentence context, definitional and synonyms clues, and optional word choices 
learned more words than did students who received definitional information only. Contextualization appears 
to be a critical component in effective vocabulary instruction, learning, and retention as postulated by the 
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National Reading Association (Butler, et al., 2010) and supported by Ahmadian, Amerian, and Goodarzi 
(2015) on Persian-speaking high school English learners. Vocabulary.com appeared to have that 
contextualized critera. 


Most Vocabulary.com participants were enrolled either in the eight-grade or ninth grade. It may be assumed 
that the grades’ proximity to when the English | EOC is commonly assessed may be a factor. The data also 
showed that 39.5 to 65.5 percent of students who participated in Vocabulary.com did not master any words. 
It is unclear how students were selected for use of the software. This is not to suggest that Vocabulary.com 
does not improve students’ word mastery because it does. In this sample, there is a high correlation 
between the number of questions answered and the number of words students mastered. It is translating 
this mastery into test performance, when all key factors are controlled, that is being evaluated in this paper. 
Moreover, because Vocabulary.com is adaptive and meets students at their level of performance, that level 
may not always coincide with the levels required to meet standards on STAAR exams. 


The study also showed that most Vocabulary.com students who made reading gains on the 2018 STAAR 
3-8 tests, either did not master any words or mastered less than 20 words. On average, students in this 
study needed to answer 18 questions correctly to master one word. Of those who answered less than 20 
questions correctly, about 99 percent did not master any words. 


As noted, it was students’ prior scores, where available, that largely predicted students’ performance on 
the 2018 STAAR 3-8 reading tests. In the regression models, students’ prior (2017 STAAR 3-8) reading 
scores were the strongest predictors of their performance. Students who came to Vocabulary.com doing 
well continued to do well. The number of students mastering more words needs to increase but this comes 
with increasing usage and the attainment of better congruence between Vocabulary.com words and words 
associated with the STAAR exams unless appropriate and alternative vocabulary measures are identified 
or developed. Vocabulary.com provided incentives for increased word mastery. 


The Vocabulary.com predictors used in this evaluation were not statistically significant in determining 
students’ reading or English | EOC performance on the 2018 STAAR tests. Where applicable, G/T was the 
only other positive predictor. All other statistically significant predictors by grade reduced students’ 
performance on the 2018 STAAR 3-8 reading and English | EOC tests. 


Overall, students who used Vocabulary.com achieved word mastery. The data in this sample showed very 
strong correlation between questions answered correctly and word mastery on Vocabulary.com. STAAR 
3-8 and EOC, as standardized tests, may not have been appropriate measures of the effectiveness of 
Vocabulary.com beyond mastery. A more appropriate measure that is congruent with the words and 
contexts in Vocabulary.com may need to be identified. 


Recommendations 


e Greater effort may have to be made to increase the number of students who consistently use 
Vocabulary.com because when disaggregated, fewer students in the sample had mastered more than 
20 words compared to those who mastered no words or less than 20 words. 


e Classroom observations and participant surveys may be required in future evaluations to determine 
how Vocabulary.com is incorporated into classroom instruction and how it is promoted to achieve the 
kind of sustained use that is necessary for word mastery since, on average, students mastered one 
word for every 18 questions answered correctly. 


e Evaluation assessments that are more compatible with the word content in Vocabulary.com may be 
required to determine its true impact on learning, given that the positive relationship between question 
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answered and word mastery is already established, but its correlation to STAAR test scores was weak 
and its predictability on STAAR was not statistically significant. 
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APPENDIX 


Table 3. Percentage of HISD Students in the Sample who Performed at or Above the 


Approaches Grade Level Standard on STAAR 3-8 Reading, 2017-2018 


Sixth Grade Seventh Grade Eight Grade 
Words Mastered n=77 ~ nApp. % App. n= 139 n App. % App. n= 1,503 n App. % App. 

0 46 32 69.6 91 66 72.5 593 434 73.2 

1-19 23 20 87.0 24 13 54.2 359 293 81.6 

20 -39 2 2 * 5 3 60.0 116 93 80.2 
40-59 2 1 id 3 * 70 55 78.6 
60-79 2 2 * 4 3 * 46 40 87.0 
80-99 - - - 1 1 * 23 19 82.6 

2100 2 1 . 11 it 63.6 296 291 98.3 

Total 77 58 67.2 139 96 76.0 1,503 1,225 81.5 


Source: Vocabulary.com; STAAR 3-8, Spring 2018 (data only) 


Table 4. Percentage Distribution of Reading Scale Score Gains by Grade Among Vocabulary.com 


Students, 2016-2017 to 2017-2018 


Wiardewmesenea Fifth to Sixth Grade Sixth to Seventh Grade Seventh to Eighth Grade 
n=77 n Gain % Gain n=139 n Gain % Gain n=1,503 n Gain % Gain 

0 46 36 78.3 91 71 78.0 593 408 68.3 
1-19 23 15 65.2 24 21 87.5 359 261 72.7 
20-39 2 4 * 5 3 60.0 116 86 74.1 
40-49 2 1 50.0 3 3 * 70 33 47.1 
60-79 a 1 * 5 2 * 46 24 52.2 
80-99 . : Z al 1 * 23 12 52.2 
2100 3 2 * 11 8 72.7 296 178 60.1 
Total 77 56 72.7 139 109 78.4 1,503 999 66.5 


Source: Vocabulary.com; STAAR 3-8 Spring 2018 (data only), STAAR regular, first time testers 
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Table 5. Multivalued Treatment Effect of Vocabulary.com Words Mastered on Sixth-Grade 


STAAR 3-8 Reading, 2017-2018 


sian a Coefficient Robust Sta. z p>z 95% Cl 

ATET 

Words Mastered Group 
(1 vs 0) 45.41 32.95 1.38 0.168 [-19.16, 109.99] 
(2 vs 0) 185.93 96.82 1.92 0.055 [-3.84, 375.70] 
(3 vs 0) 13.93 78.41 0.18 0.859 [-139.75, 167.62] 
(4 vs 0) 164.93 55.23 2.99 0.003 [56.68, 373.19] 
(6 vs 0) 9.43 126.91 0.07 0.941 [-239.31, 258.18] 

Potential Outcome Mean 

Words Mastered Group 

0 1598.07 23.03 69.4 0 [1552.93, 1643.20] 


Note: 1 = 1-19 words, 2 = 20-39 words; 3 = 40-59 words; 4 = 60-79 words; 5 = 80—99 words; and 6 = 2100 words 
Grade Level standards: Approaches: 1517-1616; Meets: 1629-1692; Masters: 1718-2056. 
STAAR regular, first-time testers 


Table 6. Multivalued Treatment Effect of Vocabulary.com Words Mastered on Seventh-Grade 


STAAR 3-8 Reading, 2017-2018 


Grade 7 Reading Robust Std. 


(n = 139) Coefficient Err. z p>z [95% Conf. Interval] 

ATET 

Words Mastered Group 
(1 vs 0) -43.42 35.71 -1.22 0.224 [-113.40, 26.57] 
(2 vs 0) -50.58 56.33 -0.90 0.369 [-160.99, 59.83] 
(3 vs 0) 65.42 76.92 0.85 0.395 [-85.35, 216.19] 
(4 vs 0) -31.33 56.84 -0.55 0.581 [-142.74, 80.07] 
(5 vs 0) 243.42 15.02 16.20 0.000 [213.97, 272.86] 
(6 vs 0) -25.76 51.91 -0.50 0.620 [-127.51, 75.98] 

Potential Outcome Mean 

Words Mastered Group 

0 1673.58 15.02 111.39 0 [1644.14, 1703.02] 


Source: Cognos 

Note: 1 = 1-19 words, 2 = 20-39 words; 3 = 40-59 words; 4 = 60-79 words; 5 = 80—99 words; and 6 = 2100 words. 
Grade Level standards: Approaches: 1567-1662; Meets: 1674-1728; Masters: 1753-2142. 
STAAR regular, first-time testers 
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Table 7. Multivalued Treatment Effect of Vocabulary.com Words Mastered on Eighth-Grade 


STAAR 3-8 Reading, 2017-2018 


aaa Coefficient eas z p>z [95% Conf. Interval] 
ATET 
Words Mastered Group 
(1 vs 0) 35.74 9.85 3.63 0.000 [16.43, 55.05] 
(2 vs 0) 43.23 15.17 2.85 0.004 [13.5, 72.96] 
(3 vs 0) 38.47 17.47 2.20 0.028 [4.23, 72.72] 
(4 vs 0) 53.76 19.27 2.79 0.005 [15.99, [91.53] 
(5 vs 0) 35.70 30.22 1.18 0.238 [-23.54, 94.94] 
(6 vs 0) 144.46 8.70 16.60 0.000 (127.41, 161.52 
Potential Outcome Mean 
Words Mastered Group 
0 1688.00 5.92 285.31 0.00 [1676.40, 1699.59] 


Note: 1 = 1-19 words, 2 = 20-39 words; 3 = 40-59 words; 4 = 60-79 words; 5 = 80-99 words; and 6 = 2100 words. 
Grade Level standards: Approaches: 1587-1691; Meets: 1700-1759; Masters: 1783-2141. 
STAAR regular, first-time testers 


Table 8. Multivalued Treatment Effect of Vocabulary.com Words Mastered on Ninth-Grade 


STAAR English | EOC, 2017-2018 


Grade oo ' Coefficient. Lancers z p>z [95% Conf. Interval] 

ATET 

Word Mastered Group 
(1 vs 0) 167.89 50.65 3.32 0.001 [68.63, 276.16] 
(2 vs 0) 264.85 61.01 4.34 0.000 (145.28, 384.42] 
(3 vs 0) 294.91 90.40 3.26 0.001 (117.74, 42.09] 
(4 vs 0) 314.00 128.89 2.44 0.015 (61.37, 566.62] 
(5 vs 0) 526.75 100.86 5.22 0.000 [329.06, 724.43] 
(6 vs 0) 687.70 60.35 11.39 0.000 [569.41, 806.00] 

Potential Outcome Mean 

Word Mastered Group 

0 3969.75 26.96 147.27 0.000 [3916.92, 4022.59] 


Note: 1 = 1-19 words, 2 = 20-39 words; 3 = 40-59 words; 4 = 60-79 words; 5 = 80-99 words; and 6 = 2100 words. 
Grade Level standards: Approaches: (2012-2015): 3750 ;3775-3976; Meets: 4000-4644; Masters: 4691-6357. 
STAAR regular, First-time testers. 
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Table 9. Selected Predictors of Performance of HISD Sixth-Grade STAAR 3-8 Reading, 


2017-2018 
Sixth Grade Reading Coefficient Beta 95% Cl 

Constant 790.69** (548.63, 1032.76] 
Words Mastered 0.15 0.05 [-0.78, 1.08] 
Questions Answered Correctly 0.00 0.01 [-0.05, 0.05] 
Economically Disadvantaged -18.37 -0.06 [-50.14, 13.39] 
At-Risk -15.16 -0.05 [-66.06, 35.75] 
Special Ed. -92.95* -0.18* [-154.81, -31.10] 
LEP -37.73 -0.10 [-87.66, 12.21] 
GIT 35.49 0.12 [-17.58, 88.56] 
Prior Reading Score 0.53** 0.62** [0.37, 0.68] 
F 56.1** 
Adjusted R? (%) 85.3 


*p < .05; *p < .001 


Table 10. Selected Predictors of Performance of HISD Seventh-Grade STAAR 3-8 Reading, 


2017-2018 

Seventh Grade Reading Coefficient Beta 95% Cl 
Constant 885.12** . (650.02, 1120.19] 
Words Mastered -0.16 -0.06 [-0.60, 0.29] 
Questions Answered Correctly 0.00 0.02 [-0.02,0.02 
Economically Disadvantaged -6.32 -0.02 [-36.39, 23.74] 
At-Risk -34.57 -0.12 [-73.99, 4.48] 
Special Ed. -18.45 -0.03 [-74.64, 37.74] 
— -57.19* -0.14* [-99.22, -15.16] 
GIT 73.27** 0.24** (35.18, 111.36] 
Prior Reading Score 0.49** 0.50** [0.35, 0.63] 
F 43.6** 
Adjusted R? (%) 71.2 


*p < .05; *p < .001 
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Table 11. Selected Predictors of Performance of HISD Eighth-Grade STAAR 3-8 Reading, 


2017-2018 
Eighth Grade Reading Coefficient Beta 95% Cl 

Constant 863.91** (788.40, 939.42] 
Words Mastered 0.02 0.02 -0.04, 0.07] 
Questions Answered Correctly 0.00 -0.02 -0.01, 0.00] 
Economically Disadvantaged -15.04* -0.05* -24.34, -5.74] 
At-Risk -48.23** -0.16** -60.62, -35.43] 
Special Ed. -66.12** -0.11** -84.75, -47.49] 
LEP -22.39* -0.05* -37.36, -7.42] 
GIT 28.44** 0.09** 17.06, 39.83] 
Prior Reading Score 0.02** 0.56** 0.48, 0.58 
F 422.7** 
Adjusted R? (%) 69.2 


*p < .05; *p < .001 


Table 12. Selected Predictors of Performance of HISD Ninth-Grade STAAR English | EOC, 


2017-2018 

Ninth Grade English | Coefficient Beta 95% Cl 
Constant 4686.36** [4641.73, 4371.00] 
Words Mastered 0.306 0.13 [-0.038, 0.651] 
Questions Answered -0.008 -0.06 [-0.028, 0.012] 
Economically Disadvantaged -151.02** -0.12** [-207.021, -95.019] 
At-Risk -697.52** -0.55** [-756.366, -638.682] 
Special Education -404.25** -0.15** [-517.193, -291.3017] 
LEP 0.306** -0.22** [-479.063, -322.691] 
F 237.8** 
Adjusted R? (%) 59.5 


*p < .05; **p < .001 
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